Papers with PEFT method
Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on parameter-efficient fine-tuning (PEFT) for dense-architecture LLMs are lacking. |
| Approach: | They propose an expert-specialized fine-tuning method that tunes the experts most relevant to downstream tasks while freezing the other experts. |
| Outcome: | The proposed method matches or surpasses full-parameter fine-tuning. |
RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have strong generalizability, but fine-tuning involves updating all parameters, rendering full fine-uning resource-intensive. |
| Approach: | They propose a parameter-efficient fine-tuning method that updates all pre-trained parameters during inference. |
| Outcome: | The proposed method outperforms baseline methods on five benchmarks across 20 datasets. |
MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have revolutionized various domains, offering unprecedented performance across numerous tasks. |
| Approach: | They propose a new Mixture of Low-Rank Experts (MoRE) for multi-task PEFT to improve performance of LLMs with fewer parameters. |
| Outcome: | The proposed method improves performance over multiple tasks and no additional inference cost. |
TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) methods have significantly reduced the number of trainable parameters needed in fine-tuning large language models. |
| Approach: | They propose a vector-based random Tensor network for high-Rank Adaptation method that achieves high-rank weight updates while retaining parameter efficiency. |
| Outcome: | The proposed method outperforms existing PEFT methods while keeping low-rank parameters. |
STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that supervised training is still necessary for complex reasoning tasks. |
| Approach: | They propose a method to integrate uncertainty-based active learning and LoRA to effectively integrate the two methods. |
| Outcome: | The proposed approach outperforms baseline models on three reasoning tasks. |
GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs (2025.acl-long)
Copied to clipboard
Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Maria Krylova, Venediktov Egor, Zuev Aleksandr, Evgeny Burnaev
| Challenge: | Recent studies show that a small subset of weights significantly impacts performance. |
| Approach: | They propose a Gaussian noise-injected fine-tuning method that updates only salient weights while injecting Gausssian into non-salient weight. |
| Outcome: | The proposed method outperforms full fine-tuning and PEFT methods under the same computational budget. |
Personalized Text Generation with Contrastive Activation Steering (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to personalized text generation rely on retrieval-augmented generation and parameter-efficient fine-tuning. |
| Approach: | They propose a training-free framework that disentangles and represents personalized writing style as a vector in LLM’s activation-space. |
| Outcome: | The proposed framework achieves 8% relative improvement in personalized generation while reducing storage requirements by 1700 over PEFT method. |
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (2022.emnlp-main)
Copied to clipboard
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao
| Challenge: | Standard fine-tuning of large pre-trained language models requires updating hundreds of millions to billions of parameters and storing a large copy of the PLM weights for every task. |
| Approach: | They propose a parameter-efficient fine-tuning technique where small trainable components are injected into the PLM and updated during fine-uning. |
| Outcome: | The proposed method outperforms SOTA parameter-efficient fine-tuning and full model fine-uning on GLUE development set with RoBERTa-large encoder. |
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models are inefficient and redundant . a light-PEFT framework can be used to prune redundant parameters during training . |
| Approach: | They propose a parameter-efficient fine-tuning framework that freezes most parameters of the foundation model and finetuns only a small number of parameters. |
| Outcome: | The proposed framework achieves training and inference speedup, reduces memory usage, and maintains comparable performance and plug-and-play feature of PEFT. |
NeuroAda: Activating Each Neuron’s Potential for Parameter-Efficient Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning are limited and require computational and memory resources. |
| Approach: | They propose a parameter-efficient fine-tuning method that enables fine-grained model finetunation while maintaining high memory efficiency. |
| Outcome: | The proposed method reduces CUDA memory usage by up to 60% while maintaining high performance. |
An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models fine-tuned with multimodal instruction-following data have demonstrated formidable capabilities in multimodal tasks. |
| Approach: | They propose to employ four PEFT methods to fine-tune the LLM component of open-source MLLMs. |
| Outcome: | The proposed method is the best performing on seven datasets, while fine-tuning the connector layers leads to improved performance in most MLLMs. |
TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition (2025.acl-long)
Copied to clipboard
Tianwei Lin, Jiang Liu, Wenqiao Zhang, Yang Dai, Haoyuan Li, Zhelun Yu, Wanggui He, Juncheng Li, Jiannan Guo, Hao Jiang, Siliang Tang, Yueting Zhuang
| Challenge: | Existing methods for fine-tuning are resource-efficient, but performance often falls short . a new approach, TeamLoRA, integrates collaborative and competitive modules to improve performance. |
| Approach: | They propose to introduce task-specific LoRA as domain experts to improve learning efficiency . teamLoRA integrates collaborative and competition modules to improve model learning . |
| Outcome: | Experiments show that TeamLoRA improves performance in multi-task learning . teamLorea integrates collaborative and competitive modules to improve performance . |
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers (2024.emnlp-main)
Copied to clipboard
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, Ivan Oseledets
| Challenge: | High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and overlook dense MLP blocks, which contain about half of the model parameters. |
| Approach: | They propose a selective PEFT method that performs well on MLP blocks by converting layer gradients into a sparse structure and reducing the number of updated parameters. |
| Outcome: | The proposed method outperforms LoRA and MeProp, robust state-of-the-art PEFT approaches. |
TARE: Lightweight Token-Aware Representation Editing for Fine-tuning Transformer-like Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing PEFT methods can be costly and underfit token-level contexts. |
| Approach: | They propose a PEFT method that performs fine-grained, token-specific edits with a small additional inference overhead and minimal tuning. |
| Outcome: | The proposed method outperforms state-of-the-art methods in 8 tasks and GLUE with a minimal tuning overhead and inference overhead. |
Low-Rank Interconnected Adaptation across Layers (2025.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method that learns weight updates W = AB for pretrained weights W through low-rank adapters A and B. |
| Approach: | They propose a low-rank interconnected adaptation across layers method that introduces an interconnected framework with locally shared A and globally shared B experts. |
| Outcome: | The proposed method improves expressiveness across domains and modalities and enables higher-rank W with equal or fewer parameters. |
Sparsity May Be All You Need: Sparse Random Parameter Adaptation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods aim at reducing computational and memory resources for fine-tuning large language models. |
| Approach: | They propose to train on a small number of parameters instead of all model parameters . they compare the method to LoRA and find it to be efficient . |
| Outcome: | The proposed method is competitive with LoRA when using a similar number of trainable parameters. |
Query-driven Relevant Paragraph Extraction from Legal Judgments (2024.lrec-main)
Copied to clipboard
| Challenge: | Legal professionals struggle with navigating lengthy legal judgements to pinpoint information that directly addresses their queries. |
| Approach: | They construct a specialized dataset to extract relevant paragraphs from legal judgements based on query . they assess the performance of current retrieval models in a zero-shot way . |
| Outcome: | The proposed model outperforms the current retrieval models in a zero-shot way and fine-tunes them using various models. |
Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models are limited due to suboptimal activation subspaces. |
| Approach: | They propose a method that leverages tail eigenvectors of model output activations to construct low-rank adapters. |
| Outcome: | The proposed method outperforms existing methods across 16 benchmarks and surpasses full fine-tuning in certain scenarios. |